Understanding Representation Learnability of Nonlinear Self-Supervised Learning

Self-supervised learning (SSL) has empirically shown its data representation learnability in many downstream tasks. There are only a few theoretical works on learnability, and of those focus final representation, treating the nonlinear neural network as ``black box". However, accurate results networks crucial for describing distribution features learned by SSL models. Our paper is first to analyze model accurately. We consider toy that contains two features: label-related feature hidden feature. Unlike previous linear setting work depends closed-form solutions, we use gradient descent algorithm train 1-layer with certain initialization region prove converges local minimum. Furthermore, different from complex iterative analysis, propose new analysis process which uses exact version Inverse Function Theorem accurately describe With this minimum, can capture at same time. In contrast, supervised (SL) learn also present processes SL via simulation experiments.